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Abstract —  Automated  systems  to  perform  aircraft 
diagnostics  and  prognostics  are  of  current  interest. 
Development  of  those  systems  requires  large  amounts  of 
data  (collection,  monitoring,  manipulation)  to  capture  and 
characterize  normal,  known  fault  events,  and  to  ensure 
data  is  captured  early  on  in  a  fault  progression  to  support 
prognostic  system  development.  Continuous  data 
collection  is  also  required  to  capture  relatively  rare 
events.  Data  collected  can  then  be  analyzed  to  assist  in 
the  development  of  automated  systems  and  for  continuous 
updating  of  algorithms  to  improve  detection, 
classification,  and  prognostic  performance.  IAC,  in 
collaboration  with  the  Air  Force  and  Army,  is  developing 
a  testbed  on  which  to  perform  data  collection,  and 
develop  diagnostic  and  prognostic  processing  techniques 
using  Army  helicopter  vibration  and  engine  performance 
data  as  part  of  the  Army’s  Vibration  Management 
Enhancement  Program  (VMEP).  VMEP  and  the  testbed 
being  developed  for  collection  and  processing  of  VMEP 
data  are  described  here. 
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1 .  Introduction 

The  Intelligent  Automation  Corporation  (IAC),  in 
collaboration  with  the  US  Army  and  Air  Force  is 
developing  a  system  for  monitoring  of  Army  helicopter 
vibration,  engine  performance,  and  aircraft  structures. 
The  processing  being  developed  is  part  of  the  Army’s 
Vibration  Management  Enhancement  Program  (VMEP). 

VMEP  is  composed  of  three  primary  components.  The 
first  component  is  an  on-board  system  that  measures  and 
processes  vibration  and  parameter  information  in  flight. 
The  second  component  is  a  ground-based  software  system 
that  displays  recommended  maintenance  actions  at  the 
aircraft,  aircraft  status  to  the  maintenance  manager,  and 


measurement  details  to  the  engineer.  The  third  component 
is  a  system  of  web-based  tools  that  provides  data 
archiving,  software  configuration  management, 
management  reports,  and  an  advanced  engineering 
development  testbed. 

IAC  supports  development  all  three  components  of 
VMEP.  In  this  paper  we  will  focus  on  the  third 
component;  the  VMEP  Server.  This  system  includes  an 
Internet  utility  to  collect  data  from  the  ground-based 
software  located  at  the  unit/aircraft  for  higher-level 
comparisons  and  statistical  modeling  as  well  as  update  of 
the  algorithms  and  parameters  of  aircraft  on-board 
systems.  All  of  the  tools  have  been  developed  using 
standard  web  server  development  software  as  well  as  the 
Mathwork’s  Matlab  and  Simulink  tools. 

Recently,  there  has  been  a  lot  of  hype  in  “prognosis.” 
Diagnostics  and  prognostic  problems  are  similar  in  nature 
[1].  Both  are  looking  for  patterns  that  are  indications  of 
faults.  Once  a  fault  pattern  is  recognized,  then  something 
needs  to  be  done  about  it.  In  diagnostic  problems,  the 
“signal-to-noise  ratio”  (SNR)  of  the  fault  signals  to  the 
ambient  background  is  high;  the  fault  is  well  developed 
and  its  signature  is  easily  seen.  And  by  definition,  a  fault 
has  already  occurred,  so  that  some  immediate 
maintenance  action  needs  to  be  taken.  For  prognosis,  the 
equivalent  fault  SNR  is  much  lower  so  that  the  fault 
signature  is  hard  to  pick  out  of  the  background.  Since  by 
definition  of  prognosis,  nothing  is  immediately  wrong, 
the  problem  becomes  prediction  of  the  time  horizon 
before  something  needs  to  be  done. 

Because  faults  in  the  prognosis  problem  appear  at  low 
SNR,  they  are  hard  to  see.  Making  accurate  predictions, 
while  maintaining  an  acceptable  false  alarm  rate,  is  much 
harder  to  do  then  with  the  diagnostics  problem.  One 
solution  is  to  integrate  (or  fuse)  low  level  signals  and 
information  that  may  be  seen  across  a  variety  of  sensors 
or  algorithms  to  effectively  increase  the  fault  SNR. 

There  is  a  problem  in  “advanced”  prognosis  where  there 
does  not  exist  good  data  sets  to  support  its  development. 
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All  the  action  in  prognosis  occurs  in  the  tails  of 
distributions  of  “normal”  vs.  “fault”  measurements. 
However  it  is  in  these  tails  that  no  data  has  ever  or  is  just 
now  starting  to  be  collected.  This  data  is  required  for 
empirical  model  development  but  also  for  validation  of 
detailed  physics  based  models.  In  current  data  sets  there  is 
lots  of  “normal”  and  “fault”  (easily  detected)  data  near 
the  mean  of  “normal”  and  “fault”.  The  tails  are  a  scary 
place.  A  slight  change  in  a  threshold  can  mean  a  drastic 
change  in  false  alarms,  missed  detections,  and  prognostic 
prediction  time  horizons. 

The  VMEP  system  and  in  particular  the  web  component 
are  ideal  for  performing  data  collection  and  algorithm 
design  and  tuning  in  order  to  develop  advanced 
diagnostic  and  prognostic  techniques  for  air  craft  health 
monitoring. 

Here  a  description  of  the  overall  VMEP  system  will  be 
given  with  emphasis  on  the  Web  Server  component.  A 
description  of  some  of  the  tools  that  have  been  developed 
and  the  results  of  their  application  to  processing  real 
Army  helicopter  data  are  presented. 

2.  Overview  of  The  Vibration 
Management  Enhancement  Program 

The  Vibration  Management  Enhancement  Program 
(VMEP)  is  a  helicopter  vibration  and  health  monitoring 
system. 


The  primary  function  of  the  VMEP  system  is  to  provide  a 
built-in  capability  to  perform  routine  vibration 
maintenance  functions  (such  as  rotor  smoothing  and 
mandatory  vibration  checks)  during  routine  operational 
flights.  In  addition,  the  system  monitors  the  status  or 
health  of  the  dynamic  drive  system  components  and 
engine  related  exceedances.  A  capability  for  flight  regime 
recognition  /  structures  monitoring  is  currently  being 
added.  The  availability  of  advanced  signal  processing  for 
machinery  fault  diagnostics  allows  much  of  the 
processing  of  vibration  signatures  and  other  monitoring 
operations  to  be  completed  during  in-flight  operation  of 
the  aircraft.  The  VMEP  is  intended  to  detect  faults  with 
sufficient  lead  time  so  that  the  ground-maintainer  can 
schedule  corrective  actions  well  before  the  fault  becomes 
an  in-flight  failure. 

Overall  system  description 

The  VMEP  system  is  composed  of  three  primary 
components.  The  first  component  is  an  on-board  system 
which  measures  and  processes  vibration  and  parameter 
information  in  flight.  The  second  component  is  a  ground- 
based  software  system  which  displays  recommended 
maintenance  actions  at  the  aircraft,  aircraft  status  to  the 
maintenance  manager,  or  measurement  details  to  the 
engineer.  The  third  component  is  a  web  server  which 
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Figure  1  The  VMEP  on-board  system 
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provides  data  archiving,  software  configuration 
management,  aircraft  status  reports,  and  an  advanced 
engineering  development  testbed.  Details  of  the  overall 
VMEP  system  can  be  found  in  [2,3]. 

On-board  system 

The  on-board  system,  shown  in  Figure  1,  consists  of  a 
Vibration  Management  Unit  (VMU),  a  wiring  harness, 
and  sensors.  The  VMU  front  panel  provides  the  aircrew  a 
method  of  selecting  acquisitions  at  specific  flight 
conditions  and  receiving  system  status  information.  The 
sensors  include  tachometers  and  accelerometers 
distributed  throughout  the  helicopter’s  drive  train. 

The  data  acquisition  process  on  board  the  VMU  is 
configurable  based  on  the  type  of  flight.  The  system  can 
be  setup  for  engineering  data  acquisition  or  for  day-to- 
day  data  collection.  An  engineering  setup  may  include 
collecting  data  in  a  raw  format  like  a  digital  tape  recorder. 
This  allows  for  the  most  flexibility  in  post  processing.  In 
normal  day-to-day  operation  the  VMEP  is  setup  to  pre- 
process  the  data  and  only  store  condensed  Condition 
Indicators  (CIs)  in  small  compact  data  files. 


If  a  new  problem  is  found  in  a  mechanical  component,  a 
small  change  in  the  setup  file  can  be  made  to  allow  the 
VMEP  to  collect  raw  or  intermediate  results  for  detailed 
engineering  analysis. 

The  data  that  is  collected  and  processed  in  the  VMU  is 
stored  for  data  transfer  after  the  aircraft  lands.  The  current 
VMU  has  96  Mbytes  of  non-volatile  memory  for  program 
and  data  storage.  A  typical  flight  contains  fewer  than  100 
Kbytes  of  data  allowing  900  typical  flights  to  be  stored 
before  the  data  needs  to  be  downloaded.  The  typical 
operation  at  a  facility  has  the  operators  downloading  the 
flights  data  at  the  end  of  the  day.  The  typical  download 
process  only  takes  one  minute.  The  size  of  the  data  files 
can  be  changed  if  engineering  desires  more  raw  data  and 
less  pre-processed  Condition  Indicators. 

Ground  based  station 

The  ground-based  software  runs  on  a  PC  based  Windows 
platform.  The  system  is  referred  to  as  the  PC  Ground- 
Based  System  (PC-GBS).  Figure  2  shows  the  opening 
screen  and  sample  selection  of  summary  reports  available 
from  the  PC-GBS.  The  operator  downloads  the  processed 
data  from  the  VMU  after  data  has  been  captured  in  flight 
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Figure  2  VMEP  PC-Ground  Based  Station  -  Example  of  summary  report  selection 
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via  a  serial  cable.  This  software  interprets  the  processed 
data  and  provides  a  multi-level  operator  interface  that  is 
oriented  to  provide  specific  data  to  assist  skilled 
maintainers  in  isolating  potential  faults. 

Where  sufficient  data  is  known  about  a  specific  fault 
indicator,  the  instructions  are  provided  for  corrective 
action.  This  software  also  allows  the  aggregation  of 
multiple  aircraft  for  comparisons  and  fleet  statistics  by 
maintenance  managers.  The  software  also  allows  detailed 
examination  of  the  data  by  engineering  personnel. 

More  importantly  for  the  work  presented  here,  the  on¬ 
board  system  and  PC-GBS  are  a  continuous  data 
collection  system.  This  data  is  automatically  downloaded 
to  the  web  component  of  VMEP  when  ever  the  PC-GBS 
is  attached  to  the  web  system. 

3 .  VMEP  Web  Server 


These  conditions  are:  nominal  operation;  operation  with 
known  faults;  and,  most  importantly  for  prognostics, 
operation  leading  up  to  the  time  that  a  fault  can  be 
detected.  Typically,  data  is  saved  only  when  a  fault  is 
detected;  too  late  to  be  useful  for  prognostics 
development. 

Figure  3  shows  the  hardware  and  connectivity  of  the 
system  used  to  perform  data  collection.  The  VMU 
collects  vibration  and  engine  performance  data. 
Maintenance  personnel  download  the  data,  via  serial  port, 
to  a  PC-GBS  which  is  typically  a  ruggidized  laptop 
computer 

Data  is  then  transferred  to  the  VMEP  Server 
automatically  when  the  PC-GBS  and  server  are 
connected.  Agent  based  software  detects  if  new  files  exist 
on  the  PC-GBS  that  do  not  exist  on  the  server.  If  not,  that 
data  is  automatically  sent. 


Connectivity 

The  VMEP  system  includes  an  internet  utility  to  collect, 
analyze,  and  make  available  data  from  the  PC-GBS 
located  at  the  unit/aircraft.  The  successful  development  of 
algorithms  for  helicopter  condition  health  monitoring 
requires  real  data  that  represent  specific  conditions. 


Users  that  do  not  have  the  PC-GBS  can  also  have  access 
to  data  and  results  using  a  standard  web  browser 
interface. 

Functional  flow  of  VMEP  Server 

Figure  4  shows  an  overview  of  the  functional  flow  of  the 
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Figure  4  VMEP  Server  functional  overview 


VMEP  Server  system.  There  are  two  major  components 
to  that  system  as  indicated  in  the  figure.  The  first  is  the 
VMEP  Server  itself  As  described  above,  the  VMEP 
Server  is  the  central  data  repository  for  collection  and 
organization  of  all  aircraft  collected  data.  This  processing 
is  handled  by  the  Automated  Data  Archive  in  the  Server. 

Network  Security  is  performed  using  a  secure  internet 
connection  and  password  protection  to  access  the  system. 
The  user  /  password  are  context  sensitive  so  that  user’s 
will  only  be  able  to  access  those  components  of  the  web 
system  for  which  they  are  authorized. 

Aircraft  Configuration  and  Software  Updates  for  the  PC- 
GBS  and  VMU  are  stored  on  the  VMEP  Server  and 
automatically  transferred  during  WEB  connection  using 
Configuration  Control.  The  updates  are  first  transferred  to 
the  PC-GBS  and  then  onto  individual  aircraft  on-board 
systems  as  opportunities  arise  (i.e.  when  maintenance 
personnel  interface  a  PC-GBS  to  a  specific  tail  number 
VMU).  Configuration  control  will  inform  maintenance  if 
too  much  time  has  passed  between  the  time  an  upgrade  is 
posted  and  it  has  not  been  transferred  to  a  specific  tail 
number.  All  the  maintenance  needs  to  do  is  attach  a  PC- 
GBS  with  the  updates  to  the  on-board  system  that  does 


not  have  the  updates. 

A  most  useful  portion  of  the  Server  for  maintenance  is  the 
Fleet  Statistics  &  Reports  section.  Here,  fleet  data, 
statistics,  trending,  and  summary  reports  are  available. 
These  reports  are  available  down  to  individual  tail 
number  and  component  level. 

Electronic  ’Help*  is  available  in  the  form  of  electronic 
user's  manuals,  power  point  training  presentations,  and 
FAQs. 

Advanced  engineering  contains  a  variety  of  modules  to 
analyze  the  incoming  data.  These  include  Diagnostics, 
Prognostics,  and  Novelty  Detection. 

Diagnostics  and  prognostics  algorithms  are  designed  to 
respond  to  known  fault  conditions.  A  novel  event  is  an 
unknown  off-nominal  condition.  That  is,  the  novel  event 
is  not  nominal  nor  is  it  classified  in  any  of  the  known 
fault  conditions.  It’s  something  completely  new. 

Novelty  detection  is  an  important  component  in  the 
operation  of  the  Server.  All  incoming  data  is  screened  to 
detect,  set  aside,  and  flag  for  engineering  analysis 
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anomaly  events.  Engineers  will  not  have  to  continuously 
examine  “normal”  events.  Rather  only  “interesting” 
events  need  be  examined.  It  is  these  sorts  of  events  that 
are  on  the  edges  of  the  data  distributions  between  normal 
and  fault  that  are  of  the  most  interest  for  developing 
prognostic  algorithms. 

The  second  component  of  Server  is  the  intelligent 
Machinery  Diagnostics  System  (iMDS)  Development 
system.  The  iMDS  Development  system  is  a  “behind  the 
scenes”  set  of  tools  used  by  diagnostic  engineers.  It 
contains  tools  for  performing  advanced  engineering 
analysis  on  data  stored  on  the  Server  and  elsewhere.  The 
toolkit  allows  engineers  to  prototype  algorithms  that  can 
later  be  incorporated  into  upgrades  for  the  PC-GBS  and 
on-board  systems. 

iMDS  Development  is  standalone  from  the  other  VMEP 
Server  system  components;  however,  it  has  the  ability  to 
download  and  process  data  from  the  iMDS  Server. 
Details  of  the  iMDS  Development  system  can  be  found  in 
[1,2]. 


4.  VMEP  Server  Operation  -  Examples 

Figure  5  shows  the  opening  screen  the  user  sees  when 
entering  the  VMEP  Server  via  the  browser  interface. 
There  are  5  major  links  from  the  home  page. 

Fleet  Analysis  is  designed  for  the  maintainer  and 
maintainer  support  personnel.  It  contains  graphical 
summaries  of  fleet  status  as  well  as  details  of  individual 
aircraft,  aircraft  component,  and  individual  condition 
indicator  (Cl)  status.  Configuration  control  of  VMEP 
software  releases  is  also  included. 

Advanced  Engineering  is  designed  for  engineers.  It 
allows  for  visualization  of  data  sets,  selection  and 
labeling  of  ‘normal’  and  ‘fault’  representative  data  sets, 
setting  of  individual  component  Cl  detection  thresholds, 
and  development  of  models  for  diagnostics,  prognostics, 
and  anomaly  detection. 
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Figure  5  VMEP  Server  Browser  Interface 
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Download  contains  the  latest  releases  of  both  the  PC- 
GBS  and  on-board  system  software  as  well  as  archived 
data  and  the  latest  VMEP  related  user  manuals. 

Support  contains  links  to  FAQ’s,  a  bug  reporting 
mechanism,  and  aircraft  (non- VMEP)  related  user’s 
manuals. 

Related  Links  contain  links  to  other  websites  that  may  be 
of  interest  to  the  user. 

Fleet  Analysis 

Whenever  possible,  visualization  of  data  processing 
results  and  summaries  have  been  used  in  the  Server.  The 
browser  interface  uses  all  the  standard  pull  down  menus, 
'back’  button,  and  hyperlinks  familiar  to  users.  'Fleet 
Analysis'  is  the  main  summary  page  used  by  maintenance 
and  fleet  maintenance  support  personnel.  Figure  6  shows 
a  sample  screen  that  is  available  from  Fleet  Overview. 


Aircraft  Status  Total  Aircraft 

AH64  52%  24%  21 


Component  Status  Total  Count 

Accessories  (  100%  20 

APU  |  100%  20 

Drive  Shafts  15%  85%  20 

Engine  1  5%  95%  20 

Engine  2  ^5%  90%  20 

Intermediate  Gearbox  [  21%  79%  19 

Main  Gearbox  5%  95%  20 

Main  Rotor  ■  35%  60%  20 

Nose  Gearbox  1  5%  95%  20 

Nose  Gearbox  2  JlO%  85%  20 

Tail  Gearbox  5%  95%  19 

Tail  Rotor  30%  55%  20 

VMU  |  35%  65%  20 

|  Exceedance  Status  |  Caution  Status  |  Good  Status 

Figure  6  Fleet  Overview  example 

The  figure  shows  at  the  top  a  summary  status  for  all 
AH64  aircraft  at  a  unit;  of  the  2 1  aircraft  being  monitored 
24%  have  at  least  one  component  that  has  a  red  (in 
exceedance)  status,  52%  are  Yellow  (or  caution)  status, 
and  24%  are  Green  (good)  status.  The  numbers  / 
percentages  shown  here  are  for  demonstration  purposes 
only.  Note  this  is  not  a  complete  data  set,  rather  selected 
tails  /  times  to  give  a  good  demonstration. 

Details  on  which  are  the  troublesome  components  can  be 
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found  in  the  bottom  portion  of  the  display.  Here  we  see 
that  the  Tail  Rotor  is  the  most  troublesome  with  15%  of 
the  aircraft  having  tail  rotor  vibrations  in  exceedance  of 
the  specified  levels.  Additional  details  can  be  found  by 
clicking  on  the  Tail  Rotor  hypertext  to  bring  up  all  of  the 
CIs  computed  for  the  tail  rotor.  A  similar  bar  graph 
summary  display  is  presented.  The  Cl  level  information 
further  isolates  problematic  areas. 

The  Fleet  Analysis  window  also  allow  maintenance  and 
maintenance  support  to  quickly  see  what  the  readiness  of 
all  /  or  specific  type  aircraft  at  a  unit  or  across  the  fleet  is. 
The  most  troublesome  components  and  the  faults 
associated  with  them  can  quickly  be  identified. 

Figure  7  shows  an  example  of  the  detail  available  when 
Aircraft  Status  by  Tail  Number  is  selected.  A  tree 
structure  similar  to  a  Windows  Explorer  tree  is  brought 
up.  That  tree  can  be  expanded  /  collapsed  to  supply  the 
user  with  the  detailed  required. 

f! 

National  Guard 
4U-B  SCARNG 
AH64 
4D-H7  UH60 

91-26326 

91-26328 

\ . -#  Absorber 

| . -©  AccGBXI 

| . -©  Ac c  GBX  2 

\ . O  Accessories 

\ . #  Engine  1 

f . ♦  Engine  2 

| . ■#  Input  GBX  1 

| . #  Input  GBX  2 

| . ■#  Inter  GBX 

| . #  Main  GBX 

I . <[)  Main  Rotor 

j  !  f . -#  Oil  Cooler 

Figure  7  Aircraft  Status  example 

Red,  yellow,  and  green  colors  are  again  used  in  the  icons 
to  indicate  the  current  status  of  all  the  aircraft  at  a  given 
point  in  the  tree.  Green  means  everything  is  within 
tolerance.  Red  means  that  some  component  is  out  of 
tolerance.  Yellow  indicates  that  the  component  is  in  a 
'warning'  band  and  requires  close  monitoring.  For 
example,  in  Figure  7  at  least  one  of  the  AH64s  has  a  red 
status  while  at  least  one  of  the  UH60  has  yellow  status. 
The  tree  has  been  expanded  to  show  that  for  a  particular 
UH60  tail  number,  the  component  that  leads  to  the 
caution  status  is  due  to  the  Main  Rotor. 


Advanced  Engineering 

Advanced  Engineering  is  intended  for  the  more 
sophisticated  engineering  user.  Currently,  Advanced 
Engineering  includes: 

o  Database  statistics 
o  Fleet  statistics  and 
o  Anomaly  detection. 

Figure  8  shows  an  example  of  the  output  obtained  when 
the  user  selects  Database  Statistics  from  the  Advanced 
Engineering  window.  The  example  shows  all  the  data  that 
has  been  stored  for  all  AH64  aircraft  to  date.  As  shown,  a 
total  of  857  data  sets  has  been  collected  which  contain 
8750  CIs  that  have  been  found  from  that  data.  This  is 
really  not  nearly  enough  if  we  want  to  use  real  data  to 
specify  a  10"5  false  alarm  threshold. 

Aircraft  type:  AH64 

..  ,  Number  of  Number  of 


BIT 

2532 

2532 

FLIGHT 

518 

23545 

GNDBAL 

157 

142 

MONITOR 

1583 

11023 

SURVEY 

136 

47 

TAIL 

166 

155 

TRACK 

178 

168 

Totals 

5270 

37612 

Fleet  Statistics  brings  up  a  display  that  compares  single 
component  -  single  Cl  for  a  particular  tail  number  to  all 
aircraft  within  the  fleet.  Similar  to  the  Aircraft  Status 
summary  tree,  Fleet  Statistics  summary  for  all  the  CIs  can 
be  obtained  through  a  similar  tree. 

On  Figure  9,  the  left  side  shows  the  Fleet  Statistics 
summary  /  selection  tree  expanded  to  highlight  specific 
tail  number  86-08996.  For  that  tail,  the  current  condition 
is  Red  (there  is  a  component  in  exceedance  that  is  not 
shown).  The  individual  components  indicate  that  Engine 
2  vibrations  are  in  the  Yellow  /  caution  limit  and  that  the 
Cl  labeled  SP2  FPG100  #2  Eng  GG  gives  rise  to  the 
caution. 

The  right  hand  side  of  Figure  9  shows  a  box  plot 
summary  of  the  selected  tail  /  component  /  condition 
indicator  compared  to  that  Cl  found  for  ah  AH64  aircraft. 

A  box  plot  summarizes  the  Cl  data.  It  indicates  the 
median,  upper  and  lower  quartile ,  upper  and  lower 
adjacent  values ,  and  outlier  individual  points.  The 
boxplot  in  Figure  9  is  broken  down  as  follows.  The  large 
dot  in  the  plot  shows  the  median  (middle  point  value)  of 
the  data.  The  ‘box’  is  drawn  so  that  50%  of  the  data  will 
reside  inside  the  box.  25%  of  the  data  is  to  the  left  of  the 
box  and  25%  is  to  the  right  of  the  box.  The  dotted  lines 
are  called  fences  or  gates.  The  gates  are  sort  of  a  poor 
man’s  single  Cl  anomaly  detector;  if  the  data  is  well 
behaved  all  the  points  should  fall  between  the  gates. 
Outlier  points  are  plotted  as  individual  circles  outside  the 
gates. 


Figure  8  Database  statistics  for  AH-64 


■B-jjfc  86-08963  H 

-EH!  86-08964 
86-08986 
-EJ-^f  86-08992 
-Eti!  86-08993 
EH$  86-08996 
El#  Accessories 
E}-#  APU 
E}<D  Drive  Shafts 
EJ-#  Engine  1 
EKI)  Engine  2 

•<D  SP2  FPG1 00  #2  Eng  GG 
O  SP2  FPG1 00  #2  Eng  PT 
O  SP4  FPG1 00  #2  ENG  OA 
EKD  Intermediate  Gearbox 


Cl  Plot 

Aircraft  Type:  AH64 
Tail  Number(s):  86-08996ALL  AH64 
Component:  Engine  2  Status:®  # 
SP2  FPG1QQ  #2  Ena  GG 


86-08996 

- 1 - 1 - 1 - 

O 

- 1 - 1 - 1 - 

i-Q-i 

- 1 - 

- 1 - 

AH64 

1—0— 1*0000 

1  1  1 

1  1  1 

1 
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O 

-0.1 

0  0.1  0.2 

0.3  0.4  0.5 

Box  Plot  -  Cl  Values 

0.6 

0.7 

0.8  0.9 

Figure  9  Advanced  Engineering  Fleet  Statistics  example 
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Aircraft  Type:  AH64 
Tail  Number:  86-08996 
Component:  Engine  2 


SP2  FPG1 00  #2  Eng  GG 


SP2  FPG1 00  #2  Eng  PT 


SP4  FPG1 00  #2  ENG  OA 
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A  . 
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Figure  10  Anomaly  Detection  example 


In  Figure  9  we  see  that  the  points  associated  with  the 
selected  tail  number  are  self  consistent.  For  the  most  part 
(with  the  exception  of  a  0  value)  they  all  lie  with  in  the 
blue  box  plus  gates.  However  it  is  easily  seen  in  the 
display  that  as  a  group,  they  are  very  different  from  the 
same  Cl  calculated  from  all  the  other  aircraft  in  the 
database. 

Anomaly  Detection 

Up  to  this  point,  each  of  the  CIs  has  been  handled  as 
single  variables.  Fusing  information  from  several  CIs  will 
take  advantage  of  the  relationships  of  CIs  in  order  to 
improve  component  level  processing. 

There  are  a  variety  of  anomaly  detectors  (ADs)  that  can 
be  used  for  the  problem  [4].  On  the  Server  we  use  a 
neural  net  solution  to  the  problem  [5].  The  basic  neural 


net  anomaly  detector  uses  radial  basis  function  (RBF) 
neural  nets  to  form  a  statistical  model  of  “nominal”  data. 
As  new  data  enters  into  the  system,  it  is  compared  to  the 
RBF  neural  net  model.  If  data  falls  within  the  boundaries 
defined  by  that  model,  then  it  is  flagged  as  “nominal”.  If 
is  does  not,  then  it  is  flagged  as  an  “anomaly”.  In  the  web 
site  several  different  anomaly  detectors  are  implemented 
using  not  only  the  RBF  neural  network,  but  Support 
Vector  Machines  and  Fuzzy  Logic.  The  user  can  call  up 
the  different  detectors  from  a  pull  down  window. 

Notice  that  there  is  an  implicit  Gaussian  assumption  in  the 
AD  as  implemented  using  the  RBF  neural  network.  In 
order  to  visualize  distribution  of  the  multi-CI  data  we 
have  created  sets  of  2-dimensional  plots  (one  for  each 
pair  of  CIs)  to  see  how  the  CIs  are  related  and  to 
determine  if  the  model  fit  is  appropriate.  Figure  10  shows 
an  example  of  that  plot  for  the  Engine  #2  on  the  AH64 
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aircraft. 


Set  Ci  Limits 


The  Engine  #2  component  has  3  CIs  associated  with  it. 
Thus  there  is  the  3  x  3  array  of  plots  shown  in  Figure  10. 
The  plots  correspond  to  pairs  of  CI  values  for  the  off 
diagonal  plots.  The  plots  along  the  main  diagonal  are 
histograms  for  the  corresponding  CI.  Note  that  the  scales 
on  the  plots  are  different  depending  on  what  is  being 
plotted.  The  CI  values  plotted  are  labeled  at  the  top  and 
sides. 

The  CI  that  was  flagged  in  the  single  CI  case 
corresponded  to  tail  number  86-08996.  Those  points 
stand  out  as  a  cluster  in  two  dimensions  in  the  figure.  For 
anomaly  detection  purposes,  those  points  were  labeled  as 
“good”  along  with  all  the  other  green  points  shown  in  the 
plot.  A  neural  net  model  was  built  using  those  green 
points.  Two  outliers,  identified  as  the  red  points,  were 
found  by  the  anomaly  detector. 

Single  CI  Detection  Threshold  Setting 

Initial  settings  for  single  CIs  on  the  Server  have  either 
been  set  based  on  the  original  manufacturers 
specifications  or  by  engineering  judgment  as  to  what  is 
acceptable  or  not.  Since  a  substantial  amount  of  data  for 
Army  aircraft  has  been  collected,  the  website  has 
included  an  automated  statistics  based  setting  for  the 
various  thresholds  of  single  CIs.  The  automated  setting  of 
thresholds  is  based  on  ordered  statistics  of  the  data. 

Figure  1 1  shows  an  example  of  that  page  for  the  AH64 
Drive  shafts  for  single  CI  threshold  setting.  There  are  four 
CIs  associated  with  that  component  but  only  two  are 
shown  in  the  figure  (the  user  would  scroll  down  on  the 
website  for  the  other  two).  The  current  and  suggested 
values  of  the  ‘goal’  (green),  ‘caution’  (yellow),  and 
‘exceedance’  (red)  are  shown. 

The  Recommended  Settings  are  found  directly  from  the 
collected  normal  data  using  the  box  plot  processing 
similar  to  that  described  before.  Figure  1 1  indicates  that 
the  Recommended  Settings  for  the  first  CI  (SP2  FPG100 
Aft  HB  Drive  Shaft),  are  considerably  above  the  current 
settings.  Indeed  this  particular  CI  was  a  problem  because 
of  the  low  threshold  settings  which  created  numerous 
false  alarms. 

As  seen  in  Figure  11,  the  user  has  the  option  to  accept 
each  of  the  recommended  settings,  fill  in  (and  accept) 
their  own  settings,  or  do  nothing. 


Aircraft  type:  AH64 

Component:  Drive  Shafts 

SP2  FPG100  AFT  HB  Drive  Shaft 


Limit  type:  Current  Settings:  Recommended  Settings:  Modified  Settings: 


Goal: 

■  15  ■ 

2.4669 

Caution: 

1.5 

2.5363 

■■ 

— ■ 

e  Accept  recommended  settings  for  this  Cl 
e  Accept  modified  settings  for  this  Cl 
<*  Accept  NOTHING!  (Make  no  changes) 


SP2FPG100AFTHB  OA 

Limit  type:  Current  Settings:  Recommended  Settings:  Modified  Settings: 


Goal:  10  3.5876  [ 

Caution:  10  3.7807  j|  H 


e  Accept  recommended  settings  for  this  Cl 
c  Accept  modified  settings  for  this  Cl 
Accept  NOTHING!  (Make  no  changes) 


Figure  11  CI  detection  threshold  setting 

To  gain  more  insight  into  the  data  the  user  may  bring  up 
additional  plots  for  the  data.  Figure  12  shows  some  of  the 
additional  plots  for  the  first  CI  for  the  Drive  Shaft.  The 
top  shows  the  boxplot  for  the  original  data.  The  bottom 
plot  is  a  histogram  of  that  data.  As  described  previously 
the  boxplot  gives  an  indication  of  the  self  consistency  of 
the  data,  the  spread  of  the  data,  and  a  poor  man’s  detector 
of  outliers.  All  the  data  that  was  used  in  plots  of  Figure  12 
had  previously  been  labeled  as  “good”. 

The  box  plots  contain  sets  up  upside  down  and  right  side 
up  triangles.  The  triangles  are  colored  green,  yellow,  and 
red  and  indicate  the  current  threshold  setting  (the  upside 
down  triangles  that  appear  above  the  center  line  in  the 
plots)  and  the  automatically  generated  thresholds  (the 
right  side  up  triangles  that  appear  below  the  center  line  in 
the  plots).  For  this  particular  CI  as  seen  all  of  the 
recommended  threshold  settings  are  to  the  right  of  the 
current  threshold  settings.  It  is  clear  from  this  plot  why 
false  alarms  would  likely  be  occurring. 

For  the  second  CI  shown  in  Figure  11  (SP2  FPG100  Aft 
HB  OA)  we  can  see  that  the  recommended  settings  are 
much  lower  then  the  current  settings.  Thus  with  the 
current  settings  there  will  be  no  false  alarms,  but  also  no 
detections!  Again  the  recommended  settings  would 
greatly  improve  the  system’s  performance. 
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Cl  Limits 


Aircraft:  AH64 
Component:  Drive  Shafts 
Cl  Name:  SP2  FPG100  AFT  HB  Drive  Shaft 


Boxplot 
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Figure  12  Cl  automated  detection  threshold 

new  data  that  enters  into  the  system.  The  data  is  the  AH64 
Engine  #2  data  that  has  been  examined  previously. 


Gold  Standard  Data  Set  Development 

As  one  can  imagine,  building  models  from  examples  of 
real  data,  requires  that  the  real  data  be  representative  of 
the  normal  and  fault  classes  that  are  being  considered. 
Miss  labeling  of  data  or  allowing  outlier  points  to  be 
included  as  “good”  points,  results  in  models  that  will  not 
be  good.  Thus  tools  have  been  included  in  the  website  for 
viewing  and  including  new  data  as  it  enters  the  system  to 
be  labeled  as  ‘normal’  or  assigning  a  specific  fault  class. 
This  is  used  in  forming  what  we’ve  called  the  gold 
standard  data  set.  All  new  data  entering  the  system  must 
be  reviewed  by  qualified  personnel  before  being  included 
in  the  gold  standard  database. 

The  Server  has  tools  built  into  it  to  be  able  to  bring  up 
new  data  for  individual  components,  display  that  data 
overlaid  with  current  gold  statndard  data.  Some  of  those 
tools  are  described  here.  The  data  and  results  presented  so 
far  have  been  on  data  that  has  been  previously  reviewed 
and  labeled  as  “good”  or  not.  That  same  data  is  presented 
below,  but  before  it  has  been  labeled. 

When  data  first  enters  the  system,  it  is  labeled  “new.”  It  is 
not  known  if  the  data  is  “normal”  or  has  some  known 
fault  condition,  or  is  just  a  bad  data  point  (i.e.  equipment 
turned  off  so  no  vibrations  generated  for  example).  It  is 
up  to  the  engineer  to  apply  a  label  to  the  data.  Figure  13 
shows  a  screen  for  reviewing  and  assigning  as  “good” 


In  the  display  shown  the  data  has  been  rank  ordered 
according  to  its  distance  [5]  from  a  single  multi¬ 
dimensional  Gaussian  model  fit  to  a  transformed  version 
of  the  multi-dimensional  Cl  data.  The  transform  converts 
the  data  from  its  original  distribution,  so  that  it  is 
Gaussian  in  each  of  its  Cl  dimensions  [6].  This 
transformation  ensures  a  better  model  fit  to  the  data.  An 
example  of  original  data  and  transformed  data  is  shown  in 
Figure  15.  Ensuring  that  the  data  and  the  form  of  the 
model  fit  are  in  agreement  is  often  overlooked  and  “bad” 
models  usually  result.  That  transformation  becomes  part 
of  the  modeling  information  and  used  for  all  future 
processing  (i.e.  the  data  is  first  transformed,  the  anomaly 
detection  of  other  processing  applied,  and  the  results  are 
transformed  back  into  the  original  domain).  Note  at  the 
bottom  of  the  display,  the  data  can  be  sorted  in  a  variety 
of  other  ways. 

Individual  points  can  be  accepted  or  rejected.  However  a 
more  automated  approach  is  to  use  the  “Rejection 
threshold”  shown  at  the  bottom  of  the  display.  Figure  14 
shows  the  results  of  a  roughly  10%  rejection  rate;  that  is 
that  the  top  10%  of  the  points  that  are  farthest  from  the 
nominal  model  are  rejected.  Those  points  are  colored  red 
in  the  plot;  accepted  points  are  green. 
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Modify  Ciass 
Aircraft:  AH64 

New  points  Component:  Engine  2 

Tail  Humber  SP2  FPG100  #2  Eng  GG  SP2  FPG100  #2  Eng  PT  SP4  FPG100  #2  EHG  QA  Date/Time_ Status 
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• 

• 
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0.01 4492  01  -Apr-2002  1 1 :57:23 
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0.011818  01 -Apr-2082  12:26:20 
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0.062366 

0.010277  11 -Apr-2002  09:12:42 

&  Accept  r  Reject 

^  Accept  changes  above  r  Accept  all  c  Reject  all  r  Rejection  threshold:Jo.300000 


Sort  by:  (Saves  changes  to  satus  column) 

r  Tail  Number  r  SP2  FPG100  #2  Eng  GG  r  SP2  FPG100  #2  Eng  PT  r  SP4  FPG100  #2  ENG  OA  r  Date/Time  Value 


Figure  13  Golden  data  set  specification 


Aircraft:  AH64 
Component:  Engine  2 
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Figure  14  Labeled  data 
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Original 


Transformed 


Figure  15  Gaussian  transformation  example 

The  histograms  along  the  diagonals  contain  red  and  green 
bars  that  correspond  to  the  red  and  green  points.  At  this 
point  the  user  can  blow  up  individual  displays,  highlight 
single  points  to  determine  which  tail  number  /  flight  that 
data  was  collected  from,  and  then  pull  up  the  data  as 
originally  collected  for  even  more  detailed  review. 


Once  the  data  has  been  validated,  the  user  pushes  a  button 
to  accept  the  labeling,  and  those  new  samples  will  be 
included  into  the  gold  standard  database.  As  more 
samples  are  collected,  the  automated  portion  of  the 
processing  will  become  more  reliable. 


Data  in  the  tails  of  the  normal  vs.  fault  distributions  will 
start  to  be  filled  in! 


prognosis  occurs  in  the  tails  of  distributions  of  “normal” 
vs.  “fault”  measurements  -  precisely  the  areas  where  not 
much  data  has  been  collected.  This  data  is  required  for 
empirical  model  development  but  also  for  validation  of 
detailed  physics  based  models.  The  tails  are  a  scary  place. 
A  slight  change  in  a  threshold  can  mean  a  drastic  change 
in  times. 

The  VMEP  system  is  an  ideal  data  collection  and 
processing  testbed  for  the  continuing  collection  and 
analysis  of  large  amounts  of  real  data  in  support  of 
automated  diagnostic  and  prognostic  system 
development.  The  VMEP  Server  included  tools  to 
automatically  screen  “normal”  and  known  fault  data  so  as 
to  detect  off-nominal  events  that  are  of  interest.  It 
includes  tools  to  systematically  continuously  update 
existing  model  parameters  to  improve  overall  detection 
and  classification  performance. 
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